Gigachad! :D

Note : Rope cache doubles the perplexity when used.

Also, quantized K cache might need to be adjusted:

If I use:

llama-perplexity -m GigaChat3-10B-A1.8B-Q8_0.gguf -mg 2 --override-kv deepseek2.expert_used_count=int:4 -c 512 -mqkv -gr -ctk q8_0 -ctv q8_0 --host 127.0.0.1 --port 8080 -f wiki.test.raw

I get:

perplexity: tokenizing the input ..
perplexity: tokenization took 325.438 ms
perplexity: calculating perplexity over 610 chunks, n_ctx=512, batch_size=2048, n_seq=4
CUDA error: invalid argument
current device: 0, in function ggml_backend_cuda_buffer_set_tensor at Q:\GitHub\ik_llama.cpp.fks\ggml\src\ggml-cuda.cu:557
cudaMemcpyAsync((char *)tensor->data + offset, data, size, cudaMemcpyHostToDevice, ((cudaStream_t)0x2))
Q:\GitHub\ik_llama.cpp.fks\ggml\src\ggml-cuda.cu:123: CUDA error

@Nexesenex

I'm only testing on CPU where I'm not seeing that error (maybe makes sense given it looks like CUDA path issue).

I don't even know what -mg 2 is but I tried with various combinations of:

    -mg 2 \
    -mqkv \
    -ger \
    --override-kv deepseek2.expert_used_count=int:2 \
    -ctk q8_0 \

I didn't use -ctv q8_0 given this is MLA attention, but seems okay. It does get "dumber" going down to only 2 experts haha...

@ubergarm : yeah, I tested with 3 experts and the PPL is not so bad. Time for inference now!

-mg is main gpu, a relic of the early llama.cpp versions used to select the GPU for mono-gpu inference, or the KV cache destination with split-row. I never knew if something else was aimed at by this command, so I set it still on my fastest GPU.

As for ctv, I always forget to remove it because it's either used, either irrelevant. :D

@Nexesenex

Note : Rope cache doubles the perplexity when used.

For the K2-Thinking too :)